Skip to content

feat(analysis): release 0.2.0 with wider injection detection and verified release governance - #4

Merged
Drako01 merged 11 commits into
mainfrom
feat/v0.2.0-audit-remediation
Nov 10, 2026
Merged

Drako01 merged 11 commits into
mainfrom
feat/v0.2.0-audit-remediation

Conversation

@Drako01

@Drako01 Drako01 commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

Summary

Release 0.2.0. Closes the gaps found in the independent engineering audit of 0.1.1 without removing or renaming any command, setting or rule code.

  • Analyzer: EXPRESSION_IN_SCRIPT now uses one shared list of untrusted contexts (github.head_ref, inputs.*, workflow_run, review, discussion, ...) and also checks format()/fromJSON() wrappers, bracket access and actions/github-script programs. New rule SELF_HOSTED_PR_REVIEW (69 codes).
  • Writes: wizard and undo resolve the target through symlinks/junctions and refuse to write outside the workspace.
  • Release governance: new governance gate step that fails if a publish Environment lacks required reviewers or a branch policy, or if the Protect main ruleset is weakened.
  • Verification: coverage floors for every measurable source folder, mutation testing for 12 modules, and real VS Code scenarios (wizard create/undo, junction refusal, diagnostics, connector with a simulated network, openFinding validation, offline AI helper).
  • Docs: security audit, threat model, release and readiness documents rewritten to match the code; single-owner approval model and rollback procedure documented; README refreshed with two new screenshots (rendered from the product HTML, labelled as such).

Repository settings already applied

The marketplace, open-vsx and github-release Environments accept deployments from main only, with no reviewer gate; secret scanning, push protection and Dependabot security updates are enabled. Merging this PR publishes 0.2.0 automatically using the pre-merge verified VSIX; no code checks rerun after merge.

Verification (local, Windows, Node 24)

  • npm run verify:enterprise: 15 of 15 steps PASS (including governance and npm audit, 0 vulnerabilities).
  • npm test: 763 pass, 0 fail, 1 skipped (real GitHub E2E, blocked without credentials).
  • npm run test:integration: PASS in real VS Code 1.145.
  • CI on final PR head 0e04ec1: SUCCESS, including Ubuntu/Windows/macOS, all 12 mutation targets and real packaged VSIX E2E. Run: https://github.com/Drako01/orquestivo/actions/runs/42871117473.

Not done / limits

  • The independent, second-person-labelled workflow corpus cannot be produced alone; the new corpus files are written by the maintainer and documented as such.
  • No manual screen-reader pass; WSL/SSH/Dev Containers checks and the real GitHub E2E remain pending. Registry-side OIDC is not verifiable from the repository.

CI remediation (2026-11-10) — sharded mutation gate

On latest commit 9e7bf03, Ubuntu no longer runs the full 12-target mutation suite inside one runner. The enterprise gate (without mutation) and 11 mutation targets run separately; write-plan.ts runs as four independent, non-overlapping 10-mutant CI shards. The REQUIRED Verify (ubuntu-latest) aggregator checks all five artifacts, ensures exactly 40 sampled write-plan mutants, applies the original 90% floor, and fails on missing/incomplete evidence. Required Windows/macOS checks are unchanged. The preview VSIX is packaged only after all three platform gates pass.

The full local verify:enterprise and verify:release commands retain their original tests. After merging, publish.yml consumes the already-approved VSIX without rerunning a quality gate.

First successful sharded mutation run: https://github.com/Drako01/orquestivo/actions/runs/42865161225

Earlier CI remediation

The original Ubuntu job was terminated (exit 143) while mutation testing; typecheck, lint, unit tests, rulesets and coverage passed, and Windows/macOS passed on the previous head.

  • Run the same 12 mutation targets with up to four isolated workers, preserving all per-target score floors and the 40-mutant limit; stop reporting a truncated target budget as PASS.
  • Stream mutation progress and permit a bounded 25-minute mutation step within CI/release job timeouts.
  • Fail closed if the governance API cannot be verified: grant read-only Actions permission and use strict enterprise verification in the PR job.
  • Commits: ae96fb0, bbad3e1, ae207ff.

Final CI run #42871117473: SUCCESS. No release or merge was performed by this remediation.

Release policy: validate once, publish tested artifact (2026-11-10)

The owner's rule is enforced on final PR head 0e04ec1 (implementation began in 3ed6a82):

  • Required Ubuntu, Windows, macOS CI gates and packaged VSIX E2E execute only on pull requests.
  • A merge to main does not re-run CI or the enterprise/release test suite.
  • publish.yml downloads the same PR-tested VSIX and archived Linux verification reports, validates successful PR CI, PR head identity, exact merged Git tree, version and SHA-256, then distributes unchanged bytes to Marketplace, Open VSX and GitHub Releases.
  • An unverified or mismatched artifact blocks publication; no untested rebuild or bypass.
  • Manual partial-publish recovery reuses a previously verified immutable artifact.

No merge was performed. Final PR head 0e04ec1 has passed the complete CI workflow.

Headless VS Code E2E remediation (2026-11-10)

After all three protected platform checks and all 12 mutation targets passed, the packaged VSIX E2E ran injectionDiagnostics: PASS but hung at the wizard scenario until the GitHub 20-minute job timeout. The repeated [AgentHost] No signed-in session lines are background VS Code logs, not assertions or a diagnosed Orquestivo failure.

Fix at 0df7bf0 plus strict TypeScript correction 0e04ec1:

  • Install deterministic modal/QuickPick/InputBox answers in Orquestivo's own extension API facade for the isolated E2E process only (env ORQUESTIVO_E2E_MODE=1); normal installed extensions register no test hooks.
  • Preserve real VSIX activation, workflow generation/undo, workspace escape/symlink refusal, diagnostics, consent, fake-network evidence, and AI checks.
  • Log START/PASS per scenario, bound each scenario to 45s, and enforce a five-minute E2E host watchdog with a clear failure reason.
  • All PR validations remain required; release after merge still reuses the verified PR VSIX without rerunning tests.

Final check run: SUCCESS — https://github.com/Drako01/orquestivo/actions/runs/42871117473. All six packaged VS Code scenarios passed, and VSIX artifact orquestivo-preview-vsix (artifact ID 13790476218) was uploaded with verified SHA-256 407154ee362f48e93a01bb84d29627350711b5e89d6189b5d7ef747549f03aee.

Test plan

  • Verify (ubuntu-latest), Verify (windows-latest), Verify (macos-latest) green on final PR head 0e04ec1
  • Package installable preview VSIX green on final PR head 0e04ec1 (six real VS Code integration scenarios under xvfb)

🤖 Generated with Claude Code

…fied release governance

Analyzer
- Detect untrusted contexts (head_ref, ref_name, inputs, workflow_run, review,
  discussion, release, client_payload) from one shared list, including inside
  format()/fromJSON(), bracket access and actions/github-script programs.
- Add SELF_HOSTED_PR_REVIEW (69 rule codes).

Security
- Resolve wizard and undo targets through symlinks/junctions and refuse to write
  outside the workspace.
- Pin that a cross-origin redirect never carries the GitHub token.

Verification
- Gate coverage for every measurable source folder, not only two facades.
- Mutation testing for 12 modules; skip type-level and message-only mutants.
- Real VS Code scenarios: wizard create/undo, junction refusal, diagnostics,
  connector with simulated network, openFinding validation, offline AI helper.
- Add release governance check (environments and ruleset) and scan benchmark.

Docs
- Rewrite security audit, threat model, release and readiness documents to match
  the code; document the single-owner approval model and rollback procedure.
- Refresh README and add two screenshots.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@Drako01 Drako01 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Todo ok

@Drako01 Drako01 self-assigned this Nov 10, 2026
@Drako01 Drako01 added the good first issue Good for newcomers label Nov 10, 2026
Drako01 and others added 8 commits November 10, 2026 12:27
The owner is the only maintainer, so a required reviewer on the publish
Environments adds a click rather than a control. Remove the reviewer gate:
merging a release to main is the approval. The Environments keep their
main-only deployment policy, and publish.yml still re-runs the strict gate
(including the governance check) before anything reaches a registry.

- Governance check: require the deployment branch policy, no longer a reviewer.
- Update release, threat model, audit, readiness and README text accordingly.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… progress

Preserve all 12 targets, the 40-mutant sampling limit and per-module score floors. Fan out across four isolated worker processes, preserve per-target evidence, fail on exceeded target budgets, and grant the enterprise gate a bounded timeout within CI and release job budgets.

Refs #4
…ctions access

Grant Actions read to the verification job so the governance gate can inspect deployment environments. Require strict results on PR CI: blocked or inconclusive evidence must not be reported green. Add a regression contract test for both controls.
Make the three OS validation checks PR-only. Reuse the exact PR-tested VSIX and archived reports on main without rerunning tests, mutation testing or builds. Validate the successful PR run, exact merged Git tree, version, source identity and SHA-256 before publication; retain explicit recovery from immutable release artifacts and fail closed on missing evidence. Add offline provenance and workflow regression tests.

Refs #4
Runner shutdown repeatedly interrupts monolithic Ubuntu mutation at write-plan 20/40. Preserve the exact 40 sampled mutants and all 12 minimum scores: run 11 normal modules separately, split write-plan into four disjoint 10-mutant jobs, then require complete shard evidence and aggregate the full write-plan 90% floor.

Keep Verify (ubuntu-latest), Verify (windows-latest), and Verify (macos-latest) as required statuses; package-preview waits for the aggregated Linux gate. Post-merge publication still reuses the approved PR VSIX without rerunning any checks. Add fail-closed aggregator tests and release documentation.

Refs #4
@Drako01

Drako01 commented Nov 10, 2026

Copy link
Copy Markdown
Owner Author

Mutation quality gate

  • Quality trend: Mutation trend not applicable
  • Quality gate: Quality gate disabled
  • Stryker is not installed locally. Probato does not install mutation dependencies automatically.

Verification (Probato PR)

Review risk: LOW · 41 file(s), +924 / -238

Findings: 0 confirmed · 2 evidenced · 0 suspected · 0 not verified

No project checks were executed.

Findings

  • LOW · Evidenced · Empty catch block in test/vscode-host.ts (test/vscode-host.ts:29)
  • LOW · Evidenced · Empty catch block in test/vscode-host.ts (test/vscode-host.ts:34)

Generated by Probato PR. Only executed checks count as proof; AI and static findings are not.

…ines

VS Code exposes distinct vscode API facades per extension, so patching window prompts in the test extension does not intercept Orquestivo's own wizard; the prompt remained pending in headless Xvfb until the 20-minute job cancellation. Install a test-only prompt bridge in Orquestivo's extension host, exercise the same real generator/undo flows, and record dialog outcomes. Add per-scenario 45s timeouts, an overall five-minute watchdog, fixture-independent regression contracts and explicit START/PASS diagnostics. Keep the packaged VSIX E2E required and do not alter post-merge publishing.

Refs #4
@Drako01
Drako01 merged commit 8bdba0e into main Nov 10, 2026
10 checks passed
Drako01 added a commit that referenced this pull request Nov 10, 2026
…ociations

GitHub Actions can return workflow_run.pull_requests=[] once a pull request has merged. Preserve strict release provenance without falsely rejecting successful exact-head CI: require merged PR identity, the expected SHA and repository, the CI workflow and event, pre-merge completion, successful status, matching Git tree, artifact manifest, version and VSIX SHA-256. Reject unrelated nonempty PR associations.

Add regression coverage reproducing PR #4 -> CI #42871117473 and document the single-validation policy. Keep v0.2.0 because no 0.2.0 release was published.
Drako01 added a commit that referenced this pull request Nov 10, 2026
…ociations (#5)

GitHub Actions can return workflow_run.pull_requests=[] once a pull request has merged. Preserve strict release provenance without falsely rejecting successful exact-head CI: require merged PR identity, the expected SHA and repository, the CI workflow and event, pre-merge completion, successful status, matching Git tree, artifact manifest, version and VSIX SHA-256. Reject unrelated nonempty PR associations.

Add regression coverage reproducing PR #4 -> CI #42871117473 and document the single-validation policy. Keep v0.2.0 because no 0.2.0 release was published.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

good first issue Good for newcomers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant